from sklearn_benchmarks.reporting.hp_match import HpMatchReporting

scikit-learn vs. ONNX Runtime

reporting = HpMatchReporting(against_lib="onnx", config="config.yml", log_scale=True)
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=brute.

WARNING! Mismatch between validation scores for 12 prediction (75.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	algorithm	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	e70d8e7886766d41f30889506baa1e67	2cf49829391aa7891e877f2ed070adf0	1.951474	0.179328	NaN	...	brute	-1	1	0.663	0.309464	0.001832	1.000	6.305983	6.306093	0.337
2	KNeighborsClassifier_brute_force	predict	100000	1	100	e70d8e7886766d41f30889506baa1e67	2cf49829391aa7891e877f2ed070adf0	0.024814	0.001929	NaN	...	brute	-1	1	1.000	18.169570	0.031776	0.757	0.001366	0.001366	0.243
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	22d5eac4a4146149b35eae6914921514	2cf49829391aa7891e877f2ed070adf0	2.721689	0.029252	NaN	...	brute	-1	5	0.757	18.296084	0.120257	0.882	0.148758	0.148761	0.125
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	16f94fc93942ff7ece860c9b0f64645f	2cf49829391aa7891e877f2ed070adf0	2.064564	0.003485	NaN	...	brute	1	100	0.882	0.314076	0.005275	1.000	6.573464	6.574391	0.118
8	KNeighborsClassifier_brute_force	predict	100000	1	100	16f94fc93942ff7ece860c9b0f64645f	2cf49829391aa7891e877f2ed070adf0	0.022841	0.000331	NaN	...	brute	1	100	1.000	18.390492	0.146075	0.757	0.001242	0.001242	0.243
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	2d5f2b2c02d77766434d9e5033b4a76c	2cf49829391aa7891e877f2ed070adf0	2.765083	0.021127	NaN	...	brute	-1	100	0.882	18.274268	0.018586	0.663	0.151310	0.151310	0.219
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	efac583623711d395ddae7a1e881ff68	2cf49829391aa7891e877f2ed070adf0	2.046789	0.002251	NaN	...	brute	1	5	0.757	0.237584	0.004532	1.000	8.614994	8.616562	0.243
14	KNeighborsClassifier_brute_force	predict	100000	1	100	efac583623711d395ddae7a1e881ff68	2cf49829391aa7891e877f2ed070adf0	0.022612	0.000737	NaN	...	brute	1	5	1.000	3.661672	0.008778	0.922	0.006175	0.006175	0.078
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	b79d63bb7670c492de5c3befac58fe29	2cf49829391aa7891e877f2ed070adf0	1.221240	0.041680	NaN	...	brute	1	1	0.663	3.709223	0.011762	0.929	0.329244	0.329246	0.266
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	e70d8e7886766d41f30889506baa1e67	88843f54689e3271092f70126e1de585	1.726383	0.025467	NaN	...	brute	-1	1	0.896	0.238396	0.003621	1.000	7.241664	7.242499	0.104
20	KNeighborsClassifier_brute_force	predict	100000	1	2	e70d8e7886766d41f30889506baa1e67	88843f54689e3271092f70126e1de585	0.003634	0.000463	NaN	...	brute	-1	1	1.000	3.647751	0.008580	0.922	0.000996	0.000996	0.078
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	22d5eac4a4146149b35eae6914921514	88843f54689e3271092f70126e1de585	2.610355	0.024594	NaN	...	brute	-1	5	0.922	3.704005	0.014146	0.896	0.704739	0.704744	0.026

12 rows × 22 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.0	6.581	0.0	-1	1	18.575	0.095	0.663	0.001	0.001	See	See
3	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.0	6.765	0.0	-1	5	0.315	0.005	1.000	0.038	0.038	See	See
6	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.0	6.596	0.0	1	100	18.665	0.145	0.882	0.001	0.001	See	See
9	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.0	6.441	0.0	-1	100	0.301	0.008	1.000	0.041	0.041	See	See
12	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.0	6.602	0.0	1	5	3.705	0.015	0.896	0.003	0.003	See	See
15	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.0	6.514	0.0	1	1	0.236	0.004	1.000	0.052	0.052	See	See
18	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.004	0.0	0.362	0.0	-1	1	3.707	0.008	0.929	0.001	0.001	See	See
21	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.004	0.0	0.367	0.0	-1	5	0.237	0.004	1.000	0.018	0.018	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.951	0.179	0.000	0.002	-1	1	0.309	0.002	1.000	6.306	6.306	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1	100	0.025	0.002	0.000	0.025	-1	1	18.170	0.032	0.757	0.001	0.001	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.722	0.029	0.000	0.003	-1	5	18.296	0.120	0.882	0.149	0.149	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1	100	0.025	0.002	0.000	0.025	-1	5	0.316	0.006	1.000	0.078	0.078	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.065	0.003	0.000	0.002	1	100	0.314	0.005	1.000	6.573	6.574	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1	100	0.023	0.000	0.000	0.023	1	100	18.390	0.146	0.757	0.001	0.001	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.765	0.021	0.000	0.003	-1	100	18.274	0.019	0.663	0.151	0.151	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.024	0.003	0.000	0.024	-1	100	0.306	0.007	1.000	0.079	0.079	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.047	0.002	0.000	0.002	1	5	0.238	0.005	1.000	8.615	8.617	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1	100	0.023	0.001	0.000	0.023	1	5	3.662	0.009	0.922	0.006	0.006	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.221	0.042	0.001	0.001	1	1	3.709	0.012	0.929	0.329	0.329	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1	100	0.022	0.000	0.000	0.022	1	1	0.237	0.004	1.000	0.091	0.091	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.726	0.025	0.000	0.002	-1	1	0.238	0.004	1.000	7.242	7.242	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1	2	0.004	0.000	0.000	0.004	-1	1	3.648	0.009	0.922	0.001	0.001	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.610	0.025	0.000	0.003	-1	5	3.704	0.014	0.896	0.705	0.705	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	2	0.006	0.002	0.000	0.006	-1	5	0.236	0.004	1.000	0.027	0.027	See	See

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=kd_tree.

WARNING! Mismatch between validation scores for 12 prediction (75.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	algorithm	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	fe8267e25dadb302ed498e50fabf7ef4	1eb6cf1a720a225efb91df0b529b0510	0.779112	1.022175	NaN	...	kd_tree	-1	1	0.929	2.626535	0.256648	1.000	0.296631	0.298044	0.071
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	fe8267e25dadb302ed498e50fabf7ef4	1eb6cf1a720a225efb91df0b529b0510	0.002450	0.000471	NaN	...	kd_tree	-1	1	1.000	120.837745	0.000000	0.946	0.000020	0.000020	0.054
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	dc74b969bf622bc24ba3bc62c980983b	1eb6cf1a720a225efb91df0b529b0510	0.989423	0.360718	NaN	...	kd_tree	-1	5	0.946	120.160118	0.000000	0.951	0.008234	0.008234	0.005
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	41accfcdcd5c50784bedf6164b63de99	1eb6cf1a720a225efb91df0b529b0510	5.436315	0.537301	NaN	...	kd_tree	1	100	0.951	2.581463	0.132271	1.000	2.105905	2.108667	0.049
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	41accfcdcd5c50784bedf6164b63de99	1eb6cf1a720a225efb91df0b529b0510	0.003278	0.000979	NaN	...	kd_tree	1	100	1.000	120.749196	0.000000	0.946	0.000027	0.000027	0.054
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	022f7445d43bb1dbc24dc3106c03cb93	1eb6cf1a720a225efb91df0b529b0510	2.904067	0.263187	NaN	...	kd_tree	-1	100	0.951	120.204526	0.000000	0.929	0.024159	0.024159	0.022
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	84622ba45e941db642965553529e1941	1eb6cf1a720a225efb91df0b529b0510	1.618465	0.201740	NaN	...	kd_tree	1	5	0.946	0.005016	0.000107	1.000	322.690709	322.763779	0.054
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	84622ba45e941db642965553529e1941	1eb6cf1a720a225efb91df0b529b0510	0.001569	0.000478	NaN	...	kd_tree	1	5	1.000	0.037146	0.000424	0.911	0.042245	0.042247	0.089
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	a9455565db1d8e052a783317c99744ff	1eb6cf1a720a225efb91df0b529b0510	0.912997	0.326837	NaN	...	kd_tree	1	1	0.929	0.060256	0.003964	0.894	15.152005	15.184764	0.035
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	fe8267e25dadb302ed498e50fabf7ef4	edf7cb014c2c0910e1c3aaaf8ae633f2	0.024846	0.013165	NaN	...	kd_tree	-1	1	0.891	0.005082	0.000127	1.000	4.889314	4.890838	0.109
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	fe8267e25dadb302ed498e50fabf7ef4	edf7cb014c2c0910e1c3aaaf8ae633f2	0.001974	0.000267	NaN	...	kd_tree	-1	1	1.000	0.037928	0.000413	0.911	0.052048	0.052051	0.089
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	dc74b969bf622bc24ba3bc62c980983b	edf7cb014c2c0910e1c3aaaf8ae633f2	0.021764	0.001449	NaN	...	kd_tree	-1	5	0.911	0.037536	0.000270	0.891	0.579805	0.579820	0.020

12 rows × 22 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	2.877	0.194	0.028	0.0	-1	1	121.443	0.000	0.929	0.024	0.024	See	See
3	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.551	0.162	0.023	0.0	-1	5	2.595	0.217	1.000	1.368	1.373	See	See
6	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.535	0.169	0.023	0.0	1	100	120.746	0.000	0.951	0.029	0.029	See	See
9	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.455	0.058	0.023	0.0	-1	100	2.603	0.190	1.000	1.327	1.331	See	See
12	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.569	0.200	0.022	0.0	1	5	0.042	0.013	0.891	84.129	87.793	See	See
15	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.426	0.035	0.023	0.0	1	1	0.005	0.000	1.000	681.254	681.333	See	See
18	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.001	0.022	0.0	-1	1	0.060	0.000	0.894	0.012	0.012	See	See
21	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.000	0.031	0.0	-1	5	0.005	0.000	1.000	0.102	0.102	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.779	1.022	0.000	0.001	-1	1	2.627	0.257	1.000	0.297	0.298	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.000	0.000	0.002	-1	1	120.838	0.000	0.946	0.000	0.000	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.989	0.361	0.000	0.001	-1	5	120.160	0.000	0.951	0.008	0.008	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	-1	5	2.639	0.298	1.000	0.001	0.001	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	5.436	0.537	0.000	0.005	1	100	2.581	0.132	1.000	2.106	2.109	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	1	100	120.749	0.000	0.946	0.000	0.000	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	2.904	0.263	0.000	0.003	-1	100	120.205	0.000	0.929	0.024	0.024	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.005	0.001	0.000	0.005	-1	100	2.598	0.187	1.000	0.002	0.002	See	See
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.618	0.202	0.000	0.002	1	5	0.005	0.000	1.000	322.691	322.764	See	See
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.000	0.000	0.002	1	5	0.037	0.000	0.911	0.042	0.042	See	See
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.913	0.327	0.000	0.001	1	1	0.060	0.004	0.894	15.152	15.185	See	See
17	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	0.000	0.001	1	1	0.005	0.000	1.000	0.214	0.214	See	See
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.025	0.013	0.001	0.000	-1	1	0.005	0.000	1.000	4.889	4.891	See	See
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	1	0.038	0.000	0.911	0.052	0.052	See	See
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.022	0.001	0.001	0.000	-1	5	0.038	0.000	0.891	0.580	0.580	See	See
23	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	5	0.005	0.000	1.000	0.379	0.379	See	See

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

WARNING! Mismatch between validation scores for 1 prediction (100.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	max_leaf_nodes	min_samples_leaf	n_iter_no_change	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	25b9f14bed7dd5d830c6ccd4dfebbf0c	8c8fffa8ec4b2d8de833421f9e32beab	0.105738	0.000401	300	...	100	100	10	0.824	0.406288	0.011753	1.0	0.260253	0.260362	0.176

1 rows × 25 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	fit	100000	100000	100	94.92	0.0	300	0.001	0.001	0.494	0.027	0.824	192.022	192.311	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.106	0.0	300	0.008	0.0	0.406	0.012	1.0	0.26	0.26	See	See